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A research design is used to find out hovr effective a 
given piece of research has been. This short course in the 
fundamentals of research design is intended to bring together a few 
typical examples and a few generally accepted prin^ciple? of research 
in a practical kit. After a brief introduction on how to progress 
from data that is uncountable to data that is countable, stress is 
placed on coming up with data that is accountable. Since analysis of 
research design must be done quickly and accurately, a shorthand 
notation is introduced that helps x-ray a wide variety of designs 
through common terminology. Whenever a piece of research has been 
analyzed in this framework, it is possible to distinguish valid 
design from invalid design. The second section of the document 
provides self-study material that can be used for independent study. 
This will give a brief introduction to the fundamentals of research 
■^des^ign,"^" (RC) — - ^ .-.-^...--^.-^ __.^....,„.^ ...... 
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This publication provides a brief introduction 
to RESEARCH DESIGN. 

After a br i ef i nt roduct i on on how to progress 
from data that is UNCOUNTABLE to data that is COUNTABLE, 
stress is placed on coming up with data that is 
ACCOUNTABLE. 

S i nee ana I ys i s of a research des i gn must be 

done quickly and accurately, a shorthand notation is 

introduced that helps x-ray a wide variety of designs 

through such common terminology as 

C - Control group 

X - Expor i emta I group 

0 - Observations or measurements 

R - Random i zat i on 

Whenever a piece of research has been analyzeci 
in this framework, it is possible to distinguish valid 
design from invalid design. 



Howard P. Alvir 
Cltire Gelifus Alvir 
27 Norwood Street 
Albany, New York 12203 
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FUNDAMENTALS OF RESEARCH DESIGN 

A research design is used to find out how effective a given piece of 
research has been. This short course in the fundamentals of research design 
is intended to give a few typical examples together with a few generally 
accepted principles of research. 

Putting these two elements together into a practical kit is the 
objective of this document. 

Even the most vague project^ objectives can eventually be reduced to 
GO FORTH AND DO GOOD I In other words, the originators of the project or 
proposal are certainly motivated by good intentions. 

On the other hand, the necessity to set up a research and evaluation 
design requires the researchers and evaluators to come up with: 

1. Countables, 

2. Quality measures, and 

3. Observable student benefits that 
document a measurable impact. 

PROGRESS TO THE TOP 

The following chart shows the progress made from UNCOUNTABLES to 
COUNTABLES to ACCOUNTABLES . The symbolic progress from to "x'' to 
is summarized on the same chart. 



% Accountables 



X 



Countables 



Uncountables 
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An example of an UNCOUNTABLE mandate is the general charge, "Go forth 
and do good I" 

An example of a COUNTABLE mandate is the charge, "Do good to at least 
25 people." 

An example of an ACCOUNTABLE mandate is the charge, "Do more good 
this year to more people than was done last year." 

Obviously, none of the above three examples is proposed as a perfectly 
formulated objective. However, each of the three example objectives is 
proposed as being much more countable and accountable than the vague uncountable 
mandate. 

PRACTICAL EXAMPLE 

As part of an evaluation exercise, teachers were told to come up with 
both SELECTION OBJECTIVES and DISSEMINATION OBJECTIVES. 

The selection objectives were to spell out the basis upon which audio-visual 
materials will be selected by specific subject matter teachers. 

The dissemination objectives are intended to spell out the basis upon 
which materials will be evaluated for effectiveness in terms of impact upon 
individual learners. 

The selection objectives are more complicated than would appear at 
first sight. For example, selecting films for a large number of different 
teachers in the same occupational area means trying to satisfy a large 
group of people, each with different interests and criteria. 
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There are other difficulties, A film that may not be appreciated by 
a specific evaluator might very well by exactly what a specific teacher is 
looking for to use with a specific group. 

Even if the evaluators are perfect from a professional judgment 
point of view, few educators want tb be told exactly what to do by an 
outside agency. 

After discussion of the above ideas, a list of selection objectives 
and a list of dissemination objectives were developed. The selection 
objectives are reprinted on page 4 • The dissemination objectives are 
reprinted on page 5 , 

Pages 7-16 provide self-study material that can 
be used for independent study. 

These pages will give a brief introduction to the 
fundamentals of research design. 



SELECTION OBJECTIVES 



AREA OF CONCERN TYPICAL OBJECTIVE SAMPLE COUNTABLE 



COMMONALITY OF AREAS Videotapes can be used in For each film, 

at least 8 out of 10 count the number 

participating BOCES of BOCES that 

use the videotape. 



COST EFFECTIVENESS 



16 mm f i Im selected is 
ava i I ab I e for dup I i cat i ng 
on V i deotape within 
a va i 1 ab I e funds 



Count the number of 
films ava liable for 
dup I i cat i ng within 
ex i st i ng funds . 



Count the number of 
f i I ms- des i red but - 
not dup I i cated 
because of excess 
cost . 



TEACHER INVOLVEMENT 



At least 3 teachers from 
at least 3 different BOCES 
f avorab I y rev i ew each 
film to be videotaped. 



Average out group 
rat i ng from the 
teacher rating form 
for each film to 
determ i ne prior i ty 
ranking for each film 



OCCUPATIONAL RELEVANCY 



At least I craft advisory 
group wi I I approve 
occupat i ona I re I evancy 
of films approved by 
teachers . 



Every film approved 
by teachers wi I I be 
subjected to approval 
by a craft adv i sory 
group us i ng the same 
rating form used by 
teachers . 
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DISSEMINATION EVALUATION 



AREA OF CONCERN 



Typica I Ob.iect i ve 



SAMPLE COUNTARLF 



AVAILABILITY 



UTILIZATION 



TEACHER NEEDS 



LEARNER NEEDS 



Each videotape will be 
made available to every 
part i c i pat i ng i nstructor 



Available videotapes will 
be utilized in the field, 



Expressed teacher needs 
will be satisfactorily met. 



Expressed learner needs 
will be satisfactorily met 
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Count the number of 
films made available 
on V i deotape. 

Count the number of 
teachers who have 
received the catalog 
of available videotapes. 

Count the number of 
showings for each 

V i deotape. 

Count the number of 
students viewing each 

V i deotape . 

Add up totals for : 
SHOWINGS and 
STUDENTS. 

Count teacher responses 
to the f o I I ow i ng 
quest i ons : 

Va I ue of film to you? 

Overa I I rat i ng of 
V i deotape? 

Would you use this 

videotape again? 

Did this videotape 
meet learners' 
needs? 



Count teacher responses 
to the fo I I ow i ng 
quest i ons : 

Number of students 
at original showing? 

at absentee showing? 

at relearning showings? 
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DISSEMINATION OBJECTIVES 



Cont . 



MAINTENANCE SURVEY 



WORK ORIENTED 
ACTIVITIES 



EVALUATION BY 
OBJECTIVES 



Adequate I og i st i ca I 
should be provided. 



support 



Mater i a I s v i deotaped are 
f undamenta I to work- 
oriented activities. 



Learner success should 
be documented by 
hard data. 



Count number of times 
usage was delayed^ 
cance I ed^ or i mpeded 
by technical breakdowns. 

Add up total dollar 
amount spent on 
repa i rs . 

Add up tota I number 
of hours consumed 
by staff on repairs. 

Count number of videotape 
documentation pages that 
specify which of the 
fol I ow i ng work-or i ented 
activities are covered 
by the videotape : 

KO (KNOWLEDGE OBJECTIVES): 
acquire occupational 

''"conterft 

PO (PERFORMANCE OBJECTIVES 
I earn a work sk i I I 

AO (ATTITUDE OBJECTIVES): 
acquire positive job 
att i tudes 

Count number of tests 
or evaluation instruments 
developed to measure 
learner success by 
hard data . 



TEACHER PREVIEW 



Each teacher shou Id be 
able to preview each 
f i Im before ut i I ization 
to determine whether the 
film can do the Job 
envisioned by the 
teacher. 



Subdivide above count 
into number of items 
measuring KO^ PO^ or 
AO as defined above. 

For each teacher^ a count 

w i I I bte kept of : 

FILMS USED and 
FILMS PREVIEWED 
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PRETEST ON 
EVALUATION DESIGNS 



DIRECTIONS: Circle the answers selected on the ANSWER SHEET provided. 

The following multiple choice question may have more 
than one required correct answer. 

Compare the answers selected with the ANNOTATED SUGGESTED 
ANSWER KEY. 



1. To develop an EVALUATION DESIGN is operationally to: 

A. Formulate statistical hypotheses which are testable formulations 
of research hypotheses. 

B. State decision rules to be followed during testing of hypotheses, 

C. Collect data according to a prespecified plan. 

D. Analyze data according to the same prespecified plan. 

E. Make decisions based upon inductive inferences concerning the 
probable truth or falsity of the research hypotheses. 

F. All of the above. 



The adequacy of an evaluation design depends upon the thoroughness 
with which the design handles the different threats to validity. 

It is usual in an experimental design to employ symbols as a type 
of shorthand for clarity of purpose and clarity of structure. 

In this designed shorthand, X is used to designate the presence of 
an experimental treatment. 

C is used to designate a control condition wherein the control is 
the absence of the experimental treatment. 

0 is used to designate an observation or measurement. 

R indicates that factors, such as selection, have been 
controlled by using randomization. 

TRUE FALSE 8. In this shorthand notation, a one-shot case study is summarized 
as X 0 which means that some treatment X has been tried on a 
single group and that an observation has been made of that group 
after the treatment. 



TRUE FALSE 2. 

TRUE FALSE 3. 

TRUE FALSE 4. 

TRUE FALSE 5. 

TRUE FALSE 6. 
TRUE FALSE 7. 
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TRUE FALSE 9. 



In the one-group pretest-posttes t format, 0X0 means that a 
pretest observation was followed by an experimental treatment which 
was then followed by a posttest observation. 



TRUE FALSE 10. In intact-group comparisons, the formulation X 0 vs. C 0 means that 
one group received the experimental treatment, that another group 
did not receive the experimental treatment, and that both groups 
were observed. 

TRUE FALSE 11. Format 1 (X 0), format 2 (0X0), and format 3 CX 0 vs^ C 0) are 
all subject to various validity threats because of inadequate 
evaluation design, 

12 • The valid evaluation designs among the following include: 

A. (X 0) 

B. (0 X 0) 

C. (X 0 vs^ C 0) 

D. (R X 0 vs_^ R C 0) 

E. (R 0 X 0 vs. R 0 C 0) 



13v The shorthand coding for posttes t-only control group design is: 

A. (X 0) 

B. (0 X 0) 

C. (X 0 vs^ C 0) 

D. (R X 0 vs^ R C 0) 

E. (R 0 X 0 vs. R 0 C 0) 

14. The shorthand notation for pretest-pos ttest conrol group design is: 

A. (X 0) 

B. (0X0) 

C. (X 0 vs^ R 0) 

D. (R X 0 vs. R C 0) 

E. (ROXOvs^ ROCO) 
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ANSWER SHEET 



1. A B C D E F 

2. TRUE FALSE 

3. TRUE FALSE 

4. TRUE FALSE 

5. TRUE FALSE 

6. TRUE FALSE 

« 

7. TRUE FALSE 

8. TRUE FALSE 

9. TRUE FALSE 
10* TRUE FALSE 
11 • TRUE FALSE 

12. A B C D E 

13. A B C D E 

14. A B C D E 



13 



10 



ANNOTATED SUGGESTED ANSWER KEY 



1 

1 • 


r 




9 


True 




Q 

^ • 


True 




4. 


True 


X 


5. 


True 


C 


6. 


True 


0 


7. 


True 


R 


8. 


True 


X 0 


9. 


True 


0X0 


10. 


True 


X 0 vs. CO 


11. 


True 


inadequate design 


12. 


D, E 


13. 


D 




14. 


E 
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POSTTEST FOR 
EVALUATION DESIGNS 



DIRECTIONS: Circle the answers selected on the ANSWER SHEET provided. 

Compare the answers selected with the ANNOTATED SUGGESTED 
ANSWER KEY. v«^i:.^i£.u 



TRUE FALSE 
TRUE FALSE 

TRUE FALSE 
TRUE FALSE 
TRUE FALSE 



TRUE FALSE 



1. A hypothesis is a tentative explanation. 

2. A hypothesis is tested by documenting empirical 
consequences . 

3. It can be safely asserted that a decision is only 
as good as the information or evidence upon which 
it is based. 

4. Decision rules developed after the empirical data 

IS collected are subject to suspicions of experimental 
bias. 

5. Data collected according to a haphazzard plan is 
subject to suspicions of manipulation. 



TRUE FALSE 6. 



TRUE FALSE 



TRUE FALSE 8. 



Statistical research often attains probable truth 
as opposed to absolute truth. 

Researchers can work with prob;ability as long as 
the researchers remain aware of the margin of error 
in even the most carefully documented experiment. 

The shorthand abbreviations used in questions 12 to 
14 on the PRETEST resemble normal everyday speech. 



9. In the Alvir evaluation design shorthand, RANDOM 
replaces R to designate the fact that the groups 
selected have been controlled by using random selection. 



TRUE FALSE 10, 



Similarily, PRE is used to stand for pretest and 
replaces 0 to designate an observation or measurement 
made before introduction of either the experimental or 
control variable. 
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TRUE FALSE 11, 



TRUE FALSE 12, 



TRUE FALSE 13 < 



TRUE FALSE 14. 

TRUE FALSE 15, 

TRUE FALSE 16, 

TRUE FALSE 17, 

TRUE FALSE 18, 

TRUE FALSE 19, 



TRUE FALSE 20. 



TRUE FALSE 21 < 



TRUE FALSE 22, 



X remains unchanged and is used to designate the 
presence of an experimental treatment or of an 
experimental -variable, 

C is unchanged and is used to designate a control 
condition wherein the control is the absence of 
the experimental treatment or of the experimental 
variable. 

POST is used to abbreviate posttest and replaces 
0 to designate an observation or measurement made 
after the experimental treatment or after the control 
condition. 

In this new Alvir shorthand for experimental design, 
the faulty design of (X 0) is replaced by X-POST. 

Any research that can be summarized by the design 
of X-POST is no longer faulty research. * 

The faulty research design of (0 X 0) is replaced 
by the notation of PRE-X-POST. 

Research designed in the format of PRE-X-POST is 
faulty because there is no control group indicated. 

The faulty research design of (X 0 vs. C 0) is 

summarized in the shorthand notation of X-POST vs. C-POST. 

The research design of X-POST vs. C-POST is correct 
because ad^^quate provision has beqn given to random 
selection both of the control group and of the 
experimental group. 

The valid research design of (R X 0 vs . R C 0) can be 
summarized in the Alvir notation by RANDOM-X-POST vs. 
RANDOM-C-POST. 

The valid evaluation design of RANDOM-X-POST vs , 
RANDOM-C-POST is less subject to sampling bias 
because the groups under experimental study and 
under experimental control have been selected at 
random. 

The valid evaluation design of (R 0 X 0 v£. R 0 C 0) 
is replaced by the Alvir shorthand notation of 
RANDOM-PRE -X-POST vs. RANDOM- PRE-C-POST. 
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TRUE FALSE 23, 



TRUE FALSE 24. 



TRUE FALSE 25. 



The evaluation design of RANDOM-PRE-X-POST vs. 
RANDOM-PRE-C-POST is dangerous for a beginneF"to 
utilize because there is no evaluation design 
provision to make sure that the randomly selected 
experimental group and randomly selected control 
group are basically equivalent as measured by a 
pretest. 

A pretest of group equivalence can only be as 
valid as the relationship of the pretest items 
to the hypothesis under investigation. 

An experimental design for a pretest which claims 
the experimental group and the control are equivalent 
because of identical weight and identical age would 
not be appropriate for hypothesis testing that refers 
to characteristics or parameters not connected to body 
bulk or to learner age. 
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ANSWER SHEET 



1 

1 • 


lKU£i 


FALSE 


o 


IKUE 


FALSE 


^ • 


TRUE 


FALSE 


A 

H- 9 


TRUE 


FALSE 


c 

D • 


TRUE 


FALSE 


o • 


TRUE 


FALSE 


/ . 


TRUE 


FALSE 


o 

o • 


TRUE 


FALSE 




TRUE 


FALSE 


1 A 


TRUE 


FALSE 


11 • 


TRUE 


FALSE 


Iz • 


TRUE 


FALSE 


IJ. 


TRUE 


FALSE 


14. 


TRUE 


FALSE 


Ij • 


TRUE 


FALSE 


Id • 


TRUE 


FALSE 


1/ • 


TRUE 


FALSE 


lo • 


TRUE 


FALSE 


l7* 


TRUE 


FALSE 


zu • 


TTT? 

IKUb 


FALSE 


21. 


TRUE 


FALSE 


22. 


TRUE 


FALSE 


23. 


TRUE 


FALSE 


24. 


TRUE 


FALSE 


25. 


TRUE 


FALSE 
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ANNOTATED SUGGESTED 
ANSWER KEY 



ANSTOR ANNOTATION 

1* True-- ^» — l^ypothesis 

2. True-- — - — — empirical consequences 

3. T;:ue — i-..^...^ — decision power 

4. True decision rules 

5. True — — — prespecified plan 

6. True probability 

7. True — — margin of error 

8. False-- ---abbreviations 

9. True RANDOM 

10. True — — — ^ pRj; 

11. True X 

12. True c 

13. True poST 

1^- True X-POST (faulty) 

15. False FAULTY 

16- True PRE-X-POST 

17. True --no CONTROL 

18- True X-POST vs. C-POST 

19. False NO RANDOMIZATION 

20. True-- RANDOM-X-POST vs. RANDOM-C-POST 

21- True - RANDOMIZATION 

22. True RANDOM- PRE -X-POST vs. RANDOM- PRE -C-POST 

23. False PRETEST EQUIVALENCE 

2^* True HYPOTHESIS CORRELATION 

25. True —RELEVANT PRETEST 
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